System and method for user monitoring and intent determination

ABSTRACT

Sensing interfaces associated with a home entertainment system are used to automate a system response to events which occur in a viewing area associated with the home entertainment system. Data derived from such sensing interfaces may also be used to enhance the response readiness of one or more system components. Still further, user presence data derived from such sensing interfaces may be used to capture and report user viewing habits and/or preferences.

BACKGROUND

Home entertainment systems comprised of plural appliances and/or thecontrolling devices used to issue commands to such appliances may beprovisioned with devices for detecting user presence and/or userinteraction via methods such as gesture, spoken voice, facialrecognition, spatial analysis, etc., as known in the art. Furthermore,growing use of personal communication devices such as smart phones,tablet computers, etc., may provide additional means for identificationof user presence via detection of such personal communication devices ona local wireless network such as a WiFi network, a Bluetooth network,etc. While multiple media sources and multiple media rendering devicesmay be coupled in many of these home entertainment systems through acentral routing appliance such as an AV receiver, set top box, smart TV,etc., no systems or methods currently exist for using user presenceand/or user interaction detection alone or in conjunction with a centralrouting appliance to provide enhanced home entertainment systemfunctionalities.

SUMMARY OF THE INVENTION

This invention relates generally to home entertainment systems andcontrol methods therefor and, in particular, to enhanced functionalitiesfor such home entertainment systems which are enabled by theavailability of additional user-related input methods for such systems.For example, in one aspect of the invention sensing interfaces such asan image sensing interface (e.g., an interface associated with acamera), a sound sensing interface (e.g., an interface associated withmicrophone), and/or an interface for sensing the presence of an RFdevice such as a smart phone may be used to fully or partially automatea system response to common events which may occur during a TV viewingsession, such as a user or users leaving or entering the viewing area, auser answering a telephone call, the detection of a doorbell ringing ora baby monitor alarm, etc. In another aspect of the invention, dataderived from such sensing interfaces may be utilized to enhance theresponsiveness of one or more system components, for example by sensingwhen a user is reaching for a physical remote control unit or preparinga component to issue a voice or gesture command. In a yet further aspectof the invention, user presence data derived from such sensinginterfaces may be used by a central routing appliance in conjunctionwith media stream information in order to capture and report userviewing habits and/or preferences.

A better understanding of the objects, advantages, features, propertiesand relationships of the invention will be obtained from the followingdetailed description and accompanying drawings which set forthillustrative embodiments and which are indicative of the various ways inwhich the principles of the invention may be employed.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various aspects of the invention,reference may be had to preferred embodiments shown in the attacheddrawings in which:

FIG. 1 illustrates an exemplary system in which the teachings of thesubject invention may be utilized;

FIG. 2 illustrates a further exemplary system in which the teachings ofthe subject invention may be utilized;

FIG. 3 illustrates, in block diagram form, an exemplary hardwarearchitecture for an appliance which may be a component part of thesystems illustrated in FIGS. 1 and 2;

FIG. 4 illustrates, in block diagram form, an exemplary softwarearchitecture for the illustrative appliance of FIG. 3; and

FIG. 5 illustrates, in flowchart form, the operation of an exemplaryevent processing module of the software architecture illustrated in FIG.4.

DETAILED DESCRIPTION

With reference to FIG. 1, an exemplary home entertainment system inwhich the methods of the subject invention may be applied may comprisean AV receiver 100 which serves as a hub for directing a selected videoand/or audio media stream from a source appliance such as, for example,a satellite or cable system set top box and/or DVR device (“STB”) 104, aDVD player 106, a CD player 108, or a game console 110 to a destinationappliance, such as a TV set 102, where the selected video and/or audiomedia stream is to be rendered. In one preferred embodiment, theconnections 130, 132 between appliances 102 through 110 and AV receiver100 may generally comprise connections for carrying HDMI-compliantdigital signals, although it will be appreciated that other interfacestandards such as component video, PCM audio, etc., may be substitutedwhere necessitated by the limitations of a particular appliance. In someembodiments, additional AV content streams may also be available from astreaming service 118 such as for example Netflix, Vudu, YouTube, NBConline, etc., via a wide area network such as the Internet 116, to whichend AV receiver 100 may be provisioned with a connection 112 to anInternet gateway device such as for example router 114. As will beappreciated, the connection between AV receiver 100 and Internet gatewaydevice 114 may be wired as illustrated, or may be wireless, e.g., a WiFilocal area network, as appropriate. To support audio and/or video phonecalling, conferencing, etc. and in accordance with certain teachings ofthe subject invention, an exemplary home entertainment system may alsobe provisioned with one or more sensing interfaces, such as interfacesassociated with microphone 120 and camera 122, suitable for the captureof audible and/or visible events within the home entertainment systemenvironment. Users 124, 126 of the illustrative entertainment system mayselect the media stream currently being viewed by means of anyconvenient method such through use of a remote control as known in theart, a voice command, a gesture, etc. In some embodiments, dataregarding these selections including without limitation media source,channel, track, title, etc., together with viewing duration, userpresence, etc. may be accumulated and reported to a database server 128for the aggregation and analysis of user viewing habits and preferencesas discussed in greater detail below.

Turning now to FIG. 2, in a second illustrative embodiment, a “smart” TVdevice 200 may incorporate both content rendering and source streamselection functionality. In this configuration, local appliances 104through 110 may be connected directly to multiple input ports of TV 200via, for example, HDMI connections 130. As is characteristic of thesmart TV genre, TV 200 may also support a connection 112 to a wide areanetwork such as the Internet 116 over which streaming AV content andother data may be received. The means for user command input to TV 200and appliances 104 through 110 may take the form of a controlling device204, for example a conventional remote control or a smart phone app incommunication with the appliances via any convenient infrared (IR),radio frequency (RF), hardwired, point-to-point, or networked protocol,as necessary to cause the respective target appliances to perform thedesired operational functions. Additionally, in certain embodiments userinput may also comprise spoken and/or gestured commands in place of orsupplemental to controlling device signals, which sounds and gesturesmay be received by microphone 120 and camera 122, processed and decodedby one of the appliances, for example TV 200, and where necessaryrelayed to other target appliances via, for example, HDMI CEC commands,IR or RF signals, etc., as described for example in co-pending U.S.patent application Ser. No. 13/657,176 “System and Method for OptimizedAppliance Control,” of common ownership and incorporated by referenceherein in its entirety.

For brevity, the discussions which follow will generally be withreference to the exemplary equipment configuration of FIG. 2, it beingunderstood however that in other embodiments, for example such thatillustrated in FIG. 1, the steps of the methods presented herein may beperformed, mutatis mutandis, by various appliances or combinations ofappliances as appropriate for a particular equipment configuration.

As illustrated in FIG. 3, an exemplary central routing appliance, suchas smart TV appliance 200, may include, as needed for a particularapplication, rendering capabilities, e.g., TV engine and media processor300 (it being appreciated that this may comprise one or more than onephysical processor depending on the particular embodiment); memory 302which may comprise any type of readable or read/write media such as RAM,ROM, FLASH, EEPROM, hard disk, optical disk, etc., or a combinationthereof; a USB interface 304; digital AV input ports and interface 306,for example DVI or HDMI; analog AV input ports and interface 308, forexample composite or component video with associated analog audio; anEthernet and/or WiFi interface 310; a Bluetooth interface 328; a digitalcamera interface 312 with associated camera 122 which may be externallyconnected or built into the cabinet of TV 200; a microphone interface314 with associated microphone 120 which may be externally connected orbuilt into the cabinet of TV 200; a remote control interface 316 forreceiving user-initiated operational commands via IR or RF signals 326;a display output 318 connected to TV screen 322; and an audio output 320connected to internal or external loudspeakers 324.

To cause the smart TV appliance 200 to perform an action, appropriateprogramming instructions may be stored within the memory 302 (hereafterthe “TV programming”) for execution by TV engine and media processor(s)300. An exemplary architecture for such TV programming is presented inFIG. 4. As illustrated, the exemplary TV programming may include, asrequired for a particular application, an underlying operating system402, such as for example LINUX, which may support a set of softwaremodules implementing the various functionalities of the smart TV device.

Such software modules may include a hardware abstraction layer 404 toprovide a device independent interface between the various applicationsoftware modules and the hardware dependent software modules such asvideo output driver 406, audio output driver 408, HDMI interface 410,analog input/output ADC/DAC 412, Ethernet and/or WiFi interface 414,Bluetooth interface 416; USB interface 418, remote control interface420, and camera and microphone drivers 422 and 423. Exemplaryapplication modules which reside above abstraction layer 404 may includeas required for a particular embodiment transport and session layerprotocol and interface management 428; AV output management 424;input/output processing and routing 440; a miscellaneous services module426 to support closed captioning, display configuration, OSD, etc.;remote control command decoder 430; and device resource management 442.Additionally, in keeping with the teachings of this invention, theexemplary TV programming may include audio and visual event detectormodules 432, 434, for example voice and/or image recognition engines orthe like; user event processing 436; and a user statistics gathering andreporting module 438.

Under the control of such TV programming, smart TV appliance 200 may forexample receive an incoming AV media stream from one of the input ports306, 308 to be processed, buffered, separated into audio and videocomponents, and routed to outputs 318, 320 for rendering on TV displayscreen 322 and loudspeaker(s) 324; may receive commands from remotecontrol interface 316 which are decoded and acted upon, for example toselect an input media stream, adjust audio volume, etc.; may manage aconnection to the Internet through Ethernet or WiFi interface 310 toenable browsing for content, download of software updates, videotelephony utilizing inputs from camera 122 and microphone 120; etc.Additionally, in accordance with the teachings herein, the exemplary TVprogramming may receive and process input signals from controllingdevice 204, camera 122 and/or microphone 120 in order to detect userpresence, identify individual users, and/or receive user command input,as will described hereafter. As will be appreciated, while in theillustrative embodiment the source of audio input signals may comprise amicrophone 120 and associated interface 314 provisioned as part of asmart TV appliance 200, in alternative embodiments audio input signalsmay be captured by any other appliance in the system and forwarded toappliance 200 for processing, or may originate from a microphoneprovisioned in a controlling device such as remote control or smartphone204, the output of which microphone may, by way of example, be digitizedand/or processed by controlling device 204 and wirelessly forwarded tosmart TV appliance 200 via remote control interface 326, WiFi interface310, Bluetooth interface 328, or any other means as appropriate for aparticular implementation.

In an exemplary embodiment the user event processing module 436 of theTV programming of TV appliance 200 (hereafter “event processing”) mayact as illustrated in the flowchart of FIG. 5 upon occurrence of auser-related event. First, at step 502 it may be determined if the eventconstitutes receipt of a remote control command, as may be reported byremote control command decoder 430. As will be appreciated, remotecontrol commands may be received via any or all of RC interface 420(e.g., infrared or RF4CE signals or the like), Ethernet/WiFi interface414, or Bluetooth interface 416, depending on the particular embodimentand the particular controlling device currently in use. If it isdetermined that the event constitutes receipt of a remote controlcommand, at step 540 the requested functional operation may be executed.By way of example without limitation, such operations may includeadjustment of output audio volume to be performed by AV managementmodule 424, selection of a new media input stream to be performed by I/Oand routing module 440, etc. In some embodiments, received remotecontrol commands may also comprise requests to direct the functionaloperation of other connected appliances, for example control of DVDplayer 106 or STB 104 via CEC commands to be issued by interfacemanagement module 428 over HDMI connections 130; or alternatively via aconnected IR blaster or a LAN as described for example in co-pendingU.S. patent application Ser. No. 13/657,176 “System and Method forOptimized Appliance Control,” of common ownership and incorporated byreference herein in its entirety. Upon completion of the requestedcommand function, at step 542 it may next be determined if the commandfunction just performed comprised a change in the media stream beingrendered by TV 200, for example selection of a new input port 306, 308;a change to the selected broadcast channel or DVR playback of STB 104; achange to an internet media source, etc. If so, at step 544 dataregarding this event may be conveyed to user statistics module 438 forlogging and ultimate reporting to database server 128. As will beappreciated, the data logged regarding the new content stream maycomprise some or all of the command parameters themselves, e.g., an STBchannel number and timestamp; metadata items obtainable from the contentsource device or channel such as a DVD title, streaming video URL, etc.;a sample of the audio or video content itself for analysis as is knownin the art and described, for example, in U.S. Pat. No. 7,986,913,7,627,477 or 7,346,512; or any other data which may be suitable foridentification purposes. If it is determined that the executed functiondid not comprise any change to the current media stream, at step 546 itmay next be determined if the executed function comprised a reportableevent, such as for example powering TV 200 (or any other connecteddevice) on or off, issuing a fast forward command during DVR playback,etc. If so, this event may also be reported to user statistics module438 for logging, after which processing of the remote control command iscomplete. In this regard, it will be appreciated that reportableappliance events such as powering an attached device on or off may alsobe separately initiated via for example direct communication to anappliance from its own remote control. Accordingly, though notillustrated, where such events are detectable, for example via HDMIstatus, absence or presence of video or audio signals, etc., theseevents may also be reported and logged.

If the reported event is not a remote control command, at step 504 theevent processing may next determine if the reported event constitutes animage change event reported by visual event detection module 432 inresponse to analysis of image data received from camera 122 via cameradriver 422. Such image processing may utilize for example the techniquesdescribed in U.S. Pat. Nos. 5,534,917, 6,829,384, 8,274,535, WIPO (PCT)patent application publication WO2010/057683A1, or the like, and may forexample periodically monitor an image comprising the field of view ofcamera 122 in order in order to initiate image analysis in response todetection of any variation in image data which exceed a certainthreshold value. If it is determined that the event is a report of adetected image change, at step 518 it is next determined if the eventcomprises the departure or imminent departure of a user from the viewingenvironment of TV 200. If so, various actions may be taken by eventprocessing 436 as appropriate. By way of example, if the departing useris the sole viewer (or, in some embodiments, the primary user, i.e. theuser who initiated the current viewing session) the event processing maybe adapted to issue a “pause” command to the source of the current mediastream. Other actions may include without limitation activating therecording function of a DVR, logging off a Web site, etc., asappropriate for a particular embodiment and configuration. If such anaction is to be taken, at step 520 in the illustrative embodiment theevent processing may cause display of a request for confirmation on TVscreen 322, e.g. “Would you like to pause this show? (Y/N).” Ifconfirmed by the user at step 522, which confirmation may take the formof a gesture, spoken command, remote control input, etc., or, in thoseembodiments where the default is to take action, a timeout, at step 528the indicated action may be executed. In this context it will beappreciated that in embodiments where voice or gesture responses areexpected, the performance accuracy of audio and/or visual eventdetection modules 432, 434 may be improved by indicating a range ofpossible responses (e.g., “yes” or “no” in this instance) to thesemodules in advance, thereby limiting the number of sound or gesturetemplates which need to be matched. Also, in some embodiments wherevoice input may be utilized, the sound level of TV audio output 320 maybe temporarily lowered to reduce background noise. Thereafter, at step530 data regarding the change in user, including user identity wherethis is determinable, for example via use of techniques such asdescribed in U.S. Pat. Nos. 7,551,756, 7,702,599, or the like, may beconveyed to statistic gathering module 438 for logging, after whichprocessing is complete.

If the detected image change event is not a user departure, at step 524the event processor may next determine if the reported event comprisesthe arrival of a new or additional user in the TV viewing environmentand if so take appropriate action. By way of example, some embodimentsmay be adapted to allow a viewer to invoke a “private viewing” statuswhich may cause the current content to be automatically muted, paused,switched, etc. in the event an additional user enters the viewingenvironment. In such embodiments, at step 526 it may be determined ifsuch a status currently exists. If so at step 528 the appropriate actionmay be executed, after which data regarding the user arrival, includinguser identity where this is determinable, may be conveyed to statisticgathering module 438 for logging, and event processing is complete. Byway of further example without limitation, re-entry of a user into aviewing environment may trigger an offer to resume playback ofpreviously paused content; or in a multi-room, multi-device household inwhich appliances are networked together and equipped with viewerrecognition, the entry of a user into one viewing environment may causethe event processor in that environment to query other event processorsand/or statistic modules elsewhere in the household to determine if thatuser has recently departed another environment, and if so, offer toresume playback of a content stream which was previously paused in thatother environment.

If the detected image change event is not a user arrival or departure,at steps 532, 534 and 536 it may next be determined if the reportedevent comprises a detectable gesture, a detectable gesture in thiscontext comprising a user action or motion which either bypre-programming or a learning process has been identified to visualevent detection module 432 as having significance as user input. If thereported event is determined to be associated with an operationalcommand function, for example “pause”, “mute”, “channel up”, etc.,processing may continue at step 540 to execute the command as describedpreviously. If the reported event is determined to be a preparatorygesture, appropriate anticipatory action may be taken at step 538. Inthis context, a preparatory gesture may comprise without limitation anypreliminary motion or gesture by a user which may be interpreted as apossible indication of that user's intent to perform an action, forexample standing up, reaching for or setting down a remote controldevice, beckoning an out of sight person to enter the room, picking up aphone, etc. Anticipatory actions may include for examplepre-conditioning visual and/or audio detection modules 432, 434 to favorcertain subsets of templates for matching purposes; modifying anon-screen menu from a format optimized for gesture control to oneoptimized for navigation via a keyboarded device such as a remotecontrol, or vice-versa; reducing or muting audio volume; signaling aremote control device exit a quiescent state, to turn on or shut off itsbacklight, or the like; etc.

If the reported event is not an image change event, at step 506 theevent processing may next determine if the reported event constitutes asound recognition event reported by audio event detection module 434.Speech or sound recognition by module 434 may utilize for example thetechniques described in U.S. Pat. No. 7,603,276, WIPO (PCT) patentapplication publication WO2002/054382A1, or the like. If the event is asound recognition event, at step 512 it may be determined if the decodedsound constitutes a voice command issued by a user. If so processing maycontinue at step 540 to execute the desired command as describedpreviously. If not a voice command, at step 514 it may be determined ifthe reported event constitutes a trigger sound. In this context atrigger sound may be an instance of a non-vocal audio signal receivedvia microphone 120 which either by pre-programming or via a learningprocess has been assigned a command or an anticipatory action. By way ofexample without limitation, trigger sounds may include a phone ordoorbell ringing, a baby monitor, smoke alarm, microwave chime, etc. Ifthe reported sound event is a trigger sound, the appropriate action,such as muting the television, etc., may be taken at step 516. Thesystem can be also be programmed to recognize various sounds and orspoken words/phrases as being indicative of a preparatory eventwhereupon one or more of the system components will be readied via ananticipatory action as described above. By way of example, a spokenphrase such as “let's see what else is on” may be recognized as apreparatory event whereupon an anticipatory action may be executed toplace a remote control device into a state wherein the remote controldevice is readied to receive input in anticipation of its use or aspoken phrase such as “come here,” the sound of a door bell, or the likemay be recognized as a preparatory event whereupon an anticipatoryaction may be executed to ready the system to look for an anticipatedevent, e.g., a specific gesture, such as a user standing up, leaving theviewing area, etc. whereupon the appropriate response action to thesensed event that was anticipated, e.g., pausing the media, may beperformed.

In the event that an anticipated event is not performed within apredetermined period of time, the system may execute, as needed, furtheractions, such as restorative action, to place the system into a state asdesired, e.g., to return one or more components of the homeentertainment system to a state where the component(s) is no longerlooking for the occurrence of the anticipation event.

If the reported event is not a sound recognition event, at step 508 theevent processing may next determine if the reported event comprises awireless device such as for example a smart phone, tablet computer, gamecontroller, etc., joining into or dropping from a LAN or PAN associatedwith smart TV 200 or other appliance in the system. If such devices havebeen previously registered with the TV programming, such activity may beused to infer the presence or absence of particular users. Suchinformation may then be processed in a similar manner to user imagedetection (e.g., processed as a user being added or departing)continuing at step 518 as previously described.

Finally, at step 510 the event processor may process any other eventreport activity as consistent with a particular embodiment. For example,in some embodiments users may be provisioned with personal remotecontrol devices which embed user identification data in their commandtransmissions, such as described in co-pending U.S. patent applicationSer. No. 13/225,635 “Controlling Devices Used to Provide an AdaptiveUser Interface,” of common ownership and incorporated by referenceherein in its entirety, in which case user presence reporting events maybe generated by remote control interface 420. In other embodimentstechnologies such as infrared body heat sensing such as proposed in theart for use in automatic unattended power-off applications may befurther adapted for the purposes described herein. Additional sources ofactivity events may also include data received from other householdequipment such as security, lighting, or HVAC control systems equippedwith occupancy sensors; entryway cameras; driveway sensors, etc., whereappropriate.

Statistic gathering module 438 may be adapted to report the dataconveyed to it during the event processing steps described above to acentralized service, e.g., hosted on Internet connected server device128, for aggregation and analysis of user viewing habits andpreferences. Depending on the particular embodiment such reporting maybe performed on an event-by-event basis, or alternatively the data maybe accumulated and reported at predetermined time intervals or only uponreceipt of a request from the server device.

By way of example, in an illustrative embodiment data reported tostatistic gathering module 438 may be formatted into several differentevent record classes and types for uploading to server device 128.Exemplary record classes may comprise user events, e.g., as may bereported at step 530 of FIG. 5; appliance events, e.g., as may bereported at step 548 of FIG. 5; and content events, e.g., as may bereported at step 544 of FIG. 5. As will be appreciated, different oradditional event classes may be appropriate in alternate embodiments andaccordingly the above classifications are presented by way of exampleonly and without limitation.

With reference to Table 1 below, user event record types may includeaddition (i.e., arrival) of user to the viewing area and deletion (i.e.,departure) of a user from the viewing area. As illustrated, each ofthese record types may include timestamp data and a user ID. Thetimestamp illustrated is suitable for use in applications where theserver device is already aware of the geographical location of thereporting system, e.g., as a result of an initial setup procedure, byURL decoding, etc. In embodiments where this is not the case, thetimestamp field may include additional data such as a time zone, zipcode, etc., where required. User identity may be any item of data whichserves to uniquely identify individual users to facilitate viewing habitand preference analysis at server 128. By way of example, user ID datamay comprise identities explicitly assigned during a setup/configurationprocess; random numbers assigned by the system as each distinct user isinitially detected; a hash value generated by a facial or voicerecognition algorithm; a MAC address or serial number assigned to asmart phone or tablet computer; etc. as appropriate for a particularembodiment.

TABLE 1 User event record Event type Time stamp User ID User:addyyyy:mm:dd:hh:mm:ss xxxxxxxx User:delete yyyy:mm:dd:hh:mm:ss xxxxxxxx

Referring now to Table 2, appliance event records may comprise recordtypes indicative of events reported to, functional commands issued to,and/or operations performed by various controlled appliances, e.g., asreported at step 548 of FIG. 5. Such events may include withoutlimitation appliance power on/off commands, playback control commands,etc., as necessary for a complete understanding of user viewing habitsand preferences. By way of example, in addition to capturing powerstatus for a DVR appliance, fast forward commands may also be logged inorder to monitor commercial skipping activity. As illustrated, eachappliance event record type may also include a timestamp field asdescribed above and an appliance type/ID field comprising an appliancetype indicator (e.g. STB/DVR, DVD, Internet stream, etc.) together witha unique ID or subtype value which may be assigned by the eventmonitoring and/or statistics gathering module in order to distinguishbetween multiple appliances of the same type, e.g., a household withmultiple DVD players, or with both cable and satellite STBs.

TABLE 2 Appliance event record Appliance Event type Time stamp type/IDAppliance:on yyyy:mm:dd:hh:mm:ss tt:xxxx Appliance:offyyyy:mm:dd:hh:mm:ss tt:xxxxx Appliance:play yyyy:mm:dd:hh:mm:ss tt:xxxxxAppliance:stop yyyy:mm:dd:hh:mm:ss tt:xxxxx Appliance:pauseyyyy:mm:dd:hh:mm:ss tt:xxxxx Appliance:ffwd yyyy:mm:dd:hh:mm:ss tt:xxxxxEtc . . .

Referring now to Table 3, content event record types may include withoutlimitation a channel or track change information record type; a titleinformation record type containing, for example, a show title retrievedfrom program guide data, DVD or video-on-demand title information, etc.;a metadata record type containing metadata values obtained from a DVD orCD, streaming video service, or the like; a content sample record typecontaining a sample clip of audio and/or video content for comparisonagainst a content identification database; or in alternate embodimentsany other data which may be utilized in determining the identity of aparticular content stream. Each record type may comprise timestamp andsource appliance fields as described above, together with a fieldcontaining identity data, which may comprise numeric, text, or binarydata as necessary. As will be appreciated, additional record and/orfield types may be utilized in other embodiments, as necessary to enablereliable identification of media content streams.

TABLE 3 Content event record Source Event type Time stamp applianceIdentity data Content:chan/track yyyy:mm:dd:hh:mm:ss tt:xxxx xxxxxContent:title yyyy:mm:dd:hh:mm:ss tt:xxxx {text} Content:metadatayyyy:mm:dd:hh:mm:ss tt:xxxx {text} Content:sample yyyy:mm:dd:hh:mm:sstt:xxxx {binary data} Etc . . .

It will be appreciated that while the exemplary data structures ofTables 1 through 3 are presented herein using a tabular format for easeof reference, in practice these may be implemented in various formsusing any convenient data representation, for example a structureddatabase, XML file, cloud-based service, etc., as appropriate for aparticular embodiment. Furthermore, it will also be appreciated thatwhile the statistics gathering and recording functionality of theillustrative embodiment is implemented as part of the programming of anexemplary appliance 200, i.e., in software module 438, in otherembodiments this functionality may be provisioned at a differentlocation, for example in one of the other appliances forming part of anentertainment system, in a local PC, at a remote server or cable systemheadend, etc., or at any other convenient location to which theparticular appliance programming may be capable of reporting userevents.

While various concepts have been described in detail, it will beappreciated by those skilled in the art that various modifications andalternatives to those concepts could be developed in light of theoverall teachings of the disclosure. For example, in alternateembodiments the steps of the methods described above may beadvantageously performed in various other appliances as appropriate,e.g. an AV receiver or a cable/satellite STB. Further, in aninterconnected system such as illustrated in FIG. 1 or 2, especiallywhere such interconnection is digital, it will be appreciated that thevarious steps of the methods may be performed by different appliances.For example without limitation, image analysis and/or speech recognitionmay be performed by a smart TV device, a locally connected personalcomputer, a home security system, etc., or even “in the cloud”, i.e., byan Internet based service, with the results reported to a user statisticgathering module and/or an event processing module resident in aconnected AV receiver or STB.

Further, while described in the context of functional modules andillustrated using block diagram format, it is to be understood that,unless otherwise stated to the contrary, one or more of the describedfunctions and/or features may be integrated in a single physical deviceand/or a software module, or one or more functions and/or features maybe implemented in separate physical devices or software modules. It willalso be appreciated that a detailed discussion of the actualimplementation of each module is not necessary for an enablingunderstanding of the invention. Rather, the actual implementation ofsuch modules would be well within the routine skill of an engineer,given the disclosure herein of the attributes, functionality, andinter-relationship of the various functional modules in the system.Therefore, a person skilled in the art, applying ordinary skill, will beable to practice the invention set forth in the claims without undueexperimentation. It will be additionally appreciated that the particularconcepts disclosed are meant to be illustrative only and not limiting asto the scope of the invention which is to be given the full breadth ofthe appended claims and any equivalents thereof.

All patents cited within this document are hereby incorporated byreference in their entirety.

1. A method for controlling at least one component device in a homeentertainment system comprised of a plurality of component devices,comprising: receiving first event data via at least one of an imagesensing interface and a sound sensing interface; determining if thereceived first event data is indicative of a preparatory event; and whenthe received first event data is determined to be indicative of apreparatory event causing an anticipatory action to be executed by atleast one of the plurality of component devices in the homeentertainment system to transition the at least one of the plurality ofcomponent devices in the home entertainment system from a first state toa second state in which the at least one of the plurality of componentdevices in the home entertainment system is readied to respond to ananticipated event wherein the anticipated event for which the at leastone of the plurality of component devices in the home entertainmentsystem is readied is an anticipated change in a number of viewers in aviewing area associated with the home entertainment system. 2.(canceled)
 3. The method as recited in claim 1, wherein the executedanticipatory action functions to transition at least one of an imagedetection module and a sound detection module from a first state into asecond state in which the at least one of the image detection module andthe sound detection module is pre-conditioned to favor one or moretemplates that are used by the image detection module and the sounddetection module to thereby ready the at least one of the imagedetection module and the sound detections module for use in determiningany second event data received via one of the image sensing interfaceand the sound sensing interface subsequent to the first event data isindicative of the anticipated event.
 4. The method as recited in claim1, comprising determining if the anticipated event occurs and, when theanticipated event is determined to have not occurred within apredetermined period of time causing a restorative action to be executedby the at least one of the plurality of component devices in the homeentertainment system whereupon the at least one of the plurality ofcomponent devices in the home entertainment system is transitioned fromthe second state into the first state in which the at least one of theplurality of component devices of the home entertainment system is nolonger readied to respond to the anticipated event.
 5. The method asrecited in claim 1, comprising determining if the anticipated eventoccurs and when the anticipated event is determined to have occurredcausing a command action to be executed whereupon at least one of theplurality of component devices is caused to perform an operationalfunction and the at least one of the plurality of component devices inthe home entertainment device is transitioned from the second state intoeither the first state or a third state in both of which the at leastone of the plurality of component devices of the home entertainmentsystem is no longer readied to respond to the anticipated event. 6.(canceled)
 7. The method as recited in claim 5, wherein the at least oneof the plurality of component devices comprises a media playback deviceand the executed command action comprises causing the media playbackdevice to perform a pause operational function.
 8. The method as recitedin claim 5, wherein the at least one of the plurality of componentdevices in the home entertainment system comprises a sound generatingdevice and the executed command action comprises causing the soundgenerating device to perform a volume control operational function. 9.The method as recited in claim 1, wherein the at least one of theplurality of component devices in the home entertainment systemcomprises a display device and the executed anticipatory actioncomprises causing a menu to be displayed on the display device.
 10. Amethod for controlling at least one component device in a homeentertainment system comprised of a plurality of component devices,comprising: receiving first event data via at least one of an imagesensing interface, a sound sensing interface, and a networked devicesensing interface; determining from the received first event data atleast one of a number of and identity of viewers in a first viewing areaassociated with the home entertainment system; when it is determinedfrom the received first event data that at least one of the number ofand identity of viewers in the first viewing area associated with thehome entertainment system has changed causing a command action to beexecuted whereupon at least one of the plurality of component devices iscaused to perform an operational function to inhibit a playing of amedia in the first viewing area associated with the home entertainmentsystem; receiving second event data via at least one of an image sensinginterface, a sound sensing interface, and a networked device;determining from the received second event data at least one of a numberof and identity of viewers in a second viewing area associated with thehome entertainment system; and when it is determined from the receivedsecond data that at least one of the number of and identity of viewersin the second viewing area associated with the home entertainment systemhas changed subsequent to at least one of the number of and identity ofviewers in the first viewing area associated with the home entertainmentsystem having changed causing a command action to be executed whereuponat least one of the plurality of component devices is caused to performan operational function to commence an unpausing and playing of themedia in the second viewing area associated with the home entertainmentsystem wherein the first viewing area and the second view area aredistinct from each other. 12-21. (canceled)
 22. A method for controllingat least one component device in a home entertainment system comprisedof a plurality of component devices, comprising: receiving event datavia at least one of an image sensing interface and a sound sensinginterface; determining if the received event data is indicative of apreparatory event; and when the received event data is determined to beindicative of a preparatory event causing a first anticipatory action tobe executed by a controlling device used to control functionaloperations of one or more of remaining ones of the plurality ofcomponent devices in the home entertainment system to transition thecontrolling device from a first state to a second state in which thecontrolling device is readied to respond to an anticipated event whereinthe anticipated event for which the controlling device is readied is ananticipated user interaction with controlling device.
 23. The method asrecited in claim 22, wherein the first state from which the controllingdevice is transitioned is a quiescent operating state of the controllingdevice.
 24. The method as recited in claim 23, wherein the second stateinto which the controlling device is transitioned is a backlit operatingstate of the controlling device.
 25. The method as recited in claim 22,comprising determining if the anticipated event occurs and, when theanticipated event is determined to have not occurred within apredetermined period of time causing a restorative action to be executedby the controlling device whereupon the controlling device istransitioned from the second state into the first state in which thecontrolling device is no longer readied to respond to the anticipatedevent.
 26. The method as recited in claim 22, comprising determining ifthe anticipated event occurs and when the anticipated event isdetermined to have occurred causing a command action to be executedwhereupon at least one of the plurality of component devices is causedto perform an operational function.
 27. The method as recited in claim22, wherein, when the received event data is determined to be indicativeof the preparatory event causing a second anticipatory action to beexecuted by one or more remaining ones of the plurality of componentdevices in the home entertainment system to transition the remaining oneor more of the remaining at least one of the plurality of componentdevices in the home entertainment system from a first state to a secondstate in which the at least one of the plurality of component devices inthe home entertainment system is readied to respond to the anticipatedevent.
 28. The method as recited in claim 27, wherein the one or moreremaining ones of the plurality of component devices in the homeentertainment system comprises a display device and the executedanticipatory action comprises causing a menu to be displayed on thedisplay device.